How does PINVOKE_OVERRIDE fit into this vs the approach in #47721? There is quite an overlap.

PINVOKE_OVERRIDE could easily be used by the native code to implement the same logic as mono_dllmap_insert used. It requires support from the host though.

I am not convinced that a managed interface is very helpful in this case. The objc_* methods themselves still have to be implemented in the native code due to the nature of the argument dispatch. Who would register the callbacks and at what time would that happen? If the answer is that it's done once at startup and driven by the host then I think the approach is unnecessarily complicated and duplicates the functionality that is already available through other means.

The PINVOKE_OVERRIDE logic was leveraged to implement this feature. The overlap here is related to implementation detail only since the PINVOKE_OVERRIDE feature isn't publicly exposed. There is no desire to presently expose that and thus we permit a narrow override for this specific scenario. I am assuming that PINVOKE_OVERRIDE is referring to what is used to handle single file work.

I am not convinced that a managed interface is very helpful in this case.

CoreCLR is not exposing functionality through native means. Using UnmanagedCallersOnly and function pointers exposing most functionality to native can be done in a bespoke manner based on a consumer's needs.

I am not convinced that a managed interface is very helpful in this case. The objc_* methods themselves still have to be implemented in the native code due to the nature of the argument dispatch.

They don't actually need to be implemented in native at all depending on what one wants to do with them. We have various tests that implement them in managed code using UnmanagedCallersOnly. This isn't to say we can handle all functionality but most is possible at least with what we've prototyped - perhaps there is an example?

Who would register the callbacks and at what time would that happen? If the answer is that it's done once at startup and driven by the host then I think the approach is unnecessarily complicated and duplicates the functionality that is already available through other means.

Yep, this must be done relatively early.

PINVOKE_OVERRIDE is for unmanaged hosts. It cannot be really replaced by managed code for all cases. The host cannot always inject the managed code soon enough for it to kick in.

Our original design goal was to make Objective-C interop work without special host. For example, to allow using libraries that use nice Objective-C interop in regular macOS console applications, with regular host that is not aware that Objective-C interop exists. It is how we have the designed the Com/WinRT interop for .NET 5 and we are happy with it.

It is not clear whether we will be able to maintain this goal due to resource constrains. If we end up giving up on this goal and make the Objective-C interop work with just the Xamarin/Apple app host, I agree that this method is not really needed. We can just use PINVOKE_OVERRIDE.

They don't actually need to be implemented in native at all depending on what one wants to do with them.

I have seen the code in the tests of the feature/objc-interop branch but I had the feeling that it covers only narrow scenario. I was thinking specifically about the implementation in Xamarin that does translation of Objective-C exceptions. I may need to read more into it though.

Our original design goal was to make Objective-C interop work without special host.

I think this somewhat slipped off my mind and it's a valid point. On the other hand it feels like SetMessageSendCallback specifically is problematic if two different libraries tried to use it. Likely not a big deal in practice but I remember seeing the concern in some comment earlier.

I was thinking specifically about the implementation in Xamarin that does translation of Objective-C exceptions.

If one wants exceptions support, I agree that is likely not possible to implement in C#. However that should not be the reason for forcing it be a hard requirement.

On the other hand it feels like SetMessageSendCallback specifically is problematic if two different libraries tried to use it.

Yes that is true. This API is really a niche case for Xamarin at this point. The ComWrappers also has an issue with singleton usage - the GC interaction model. It was a desire to make that more narrow but given Xamarin established guidance around usage of P/Invoke with Selectors it is hard to have a model that can work with multiple consumers.

I guess my point was that if it's primarily for the Xamarin use-case then PINVOKE_OVERRIDE would work there just as good. It has the additional benefit of being slightly more flexible and guaranteeing the registration soon enough to ensure that all P/Invokes are consistently resolved (and thus it's easier to debug and reason about). Since there can only be one host it also takes care of the issue of two consumers trying to use the interface. Lastly, it's available on Mono and CoreCLR today so the implementation for both runtimes could be easily shared on the Xamarin side.

Obviously that was all under the assumption that one would have the control of the unmanaged host. If this is supposed to cover other scenarios like running under the default CoreCLR host (corehost, corerun or other) and possibly a managed implementation of the callbacks then I see the reason for the different approach.

I am still unsure what can you actually achieve in the managed code and how would you handle all the cases of parameter passing and forwarding that to the underlying implementation. The unit test doesn't seem to cover that ground.